Technology and Techniques for Conducting Instant Messaging Studies

Designing usability lab tests of instant-messaging services, whether conducted on a hand-held device or a computer, presents unique challenges for the testing team. This presentation describes three instant-messaging studies and the technology and techniques used to instill realism and maintain rigor. INTRODUCTION Instant messaging, or IM, is a popular consumer and business tool that offers “real-time synchronous collaboration. Real-time collaboration tools allow geographically dispersed people to “meet online” [1]. Now, thanks to easy (often free) access to IM software and the proliferation of mobile phones, as well as recent strides in the capabilities of both, a whole new IM “patois” has emerged, making IM hugely popular with younger adults leading mobile lifestyles. “IMing” also has become a desirable way for computer users—both personal and business—to get quick responses [2] to messages, knowing when their colleagues or friends are online and available to converse. In July 2000, there were an estimated over 100 million IM users, over 12 million of them business IM users [2]; we can only assume those numbers have grown in the last five years. The authors have designed and conducted lab studies of hand-held and computer instant-messaging services over the past 10 years, beginning with two-way pagers used mainly by early adopters. Both authors are senior-level usability researchers who have conducted scores of studies—including over a dozen mobile and/or instant-messaging studies— with hundreds of users. This paper describes three of the authors’ recent lab studies of instant-messaging and video instant-messaging services and the technology and techniques used to instill realism and maintain rigor. Note that we will not discuss the merits of specific brands of equipment or IM software. Rather, our paper focuses on ways to plan and set up a lab study for conducting IM tests using realistic scenarios. CHALLENGES OF TESTING REAL-TIME COMMUNICATION REALISTICALLY Designing usability lab tests of instant-messaging services, whether conducted on a hand-held device, a computer, or both, presents unique challenges for the testing team. At a high-level, testing real-time IM communication is unlike testing “traditional” websites or software, because IM studies require that test users and test facilitators be interactive with each other, not just the computer. In IM studies, we need to evaluate not only users interacting with a computer, but also the reactions that users have with their IM partners [1]. Test facilitators have to be able and willing to take on the roles of facilitator, observer, and interactive participant. We also have to devise scenarios that are realistic enough to motivate people to engage in a conversation and be comfortable doing so.